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Executive Summary 

In this report on the state of the research and development of instructional simulation we have 
responded to 12 questions posed by the DSMC staff. These questions are included in three categories: 
The constituents of good instructional simulations, the design and development of instructional 
simulations, and the evaluation and testing of instructional simulation. Our work is based primarily 
on careful review and analysis of the extensive literature on the topic supplemented by interviews 
with developers and users. 

Highlights of the report are as follows: 

o Emphasis on the need for a greater empirical approach in simulation 
design. 

o Recommended set of phases for using instructional simulations which can 
serve as an instructional simulation users guide to increase learning from 
simulations* 

o Detailed comparison of the cost/benefits of simulation compared to other 
methods. 

o Discussion of the issues relevant to fidelity between the simulation and 
what is represented. 

o Conditions which determine the instructional value of verisimilitude or 
fidelity are enumerated. 

o Design steps are listed and discussed. 

o Emphasis on task, cognitive, or affective analysis as an early phase of 
simulation design. 

o Discussion of the advantages and disadvantages of task, cognitive, or 
affective analysis during simulation design. 

o Overview of instructional effects research. 
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Frame or checklist for conducting a content analysis to evaluate the 
potertial of an instructional simulation is presented. 

Step by step procedure for conducting local empirical evaluations of 
learning from simulations. 

Appendix devoted to special considerations of computers in instructional 
simulation. 



Introduction 



Our purpose is to communicate our findings on current knowledge of simulation design 
development and evaluation. More specifically, we will discuss what currently is seen to constitute 
good simulation design, development, and practice. Throughout our research toward this purpose we 
were guided by a set of questions provided by DSMC personnel. These specific questions are 
incorporated in the body of this report as second order headings under three first order headings. 

A study of instructional simulation cannot be isolated from the study of other instructional 
methods. Answers to relevant questions and responses to critical issues must be created in the 
matrix of comparisons and contrasts of other instructional methods and the designing of other types of 
instruction. In our view some of the problems attendant to the research and development literature 
of simulation and instructional simulation are functions of the failure to address similarities of issues 
and design strategies between other methods and that of instructional simulation and the failure to 
engage those similarities. 

This is not to be taken to mean that we imply that there are not unique issues characteristic of 
design of instruction in other methods and instructional simulation. The probability is that there are 
some differences which must be honored among the various methods of instruction and the design of 
instruction within the rubrics of those instructional methods. We view instructional design as an 
applied science, albeit an applied social science, and one heart of any science is deep analysis of 
similarities and differences; and, for best application, "best management" of similarities and differences 
are essential. 

The subsequent sections of this report are, first, a brief report on our methods. We follow this 
with a short overview of instructional methods other than simulation which will serve as a basis for 
later comparisons and contrasts. This is followed by three major topical headings: the constituents of 
good instructional simulations; the design and development of instructional simulations; and the 
evaluation and testing of instructional simulations. Following comes the conclusions, references, and 
appendices consisting of (A) a special short paper on computers in instructional simulation: (B) an 
extensive bibliography of items examined, (many of which were not cited in the body of the report); 
and (C) the interview guideline and the credentials of interviewees. 

Methods 

In this report we will record the results of a two dimensional effort. First, the report is based on 
an extensive review of the instructional simulation literature and to a lesser degree the instructional 
design literature. Second, the report is based on interviews of several individuals who develop and 
use instructional simulations. We think that this two dimensional approach is necessary for the 
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construction of a record displaying the state of instructional simulation. In this area, perhaps more 
so than others in instruction, there are at least these two sources of knowledge. The extensive 
literature is, of course, a record of those in the field who not only design and use instructional 
simulation, but also a record of their findings and impressions in the professional literature. 

Our review of the literature included use of the following reference aids: LCS (an on-line computer 
reference system); ERIC; INFO TRACK; EjtoiiojiJnifix; SflfiJfll Science Citation Inde?L .(Cttaiifln_and 
Source volumes); and Eojj^iiLErini. In order to search the above reference aids, various 
combinations of the following key terms were used for the subject search: simulation games in 
education; educational games; simulated environment and teaching method; instructional simulation 
(and teaching methods, and study and teaching, and debriefing, and design, and models, and 
techniques, and heuristics, and learning effects, and evaluation); simulation methods; education and 
simulation methods; synthetic training devices; microteaching; cognitive task analysis; and affective 
task analysis. After reviewing relevant sources and their associated bibliographies additional searches 
were conducted for specific authors or titles cited. 

In addition to the literature search and analysis we conducted interviews with individuals who 
design and use instructional simulations; but, who for a variety of reasons, do not publish routinely in 
the area of instructional simulation. Because we know that these individuals have considerable 
knowledge we have interviewed as many as our resources have allowed. We feel that their 
experiences are valid and important resources in the same sense that there are many who plan and 
give great lectures who have not contributed to the literature on the lecture as an instructional 
method and who could, nevertheless, provide considerable insight into instruction using that method. 

Throughout our search we were guided by questions posed by the staff at DSMC; but, as we 
progressed, other questions emerged to which we feel bound to respond. These we have incorporated 
into assorted sections. These questions, taken together, have become the organizer for this report. 

In our search we are impressed with the extensive ness of the literature. The uses of simulation 
extend as follows: into most disciplines; into most levels of instruction from elementary to adult; 
across several strategies, or simulation and game types [unregimented play, simple role playing, 
systems games, simulations relying on predetermined roles or action cards, person-computer, reactions 
simulations, and monopoly-type box games (Davis, 1930)]; and into practically all the intended 
outcomes or objectives of education in the three traditional domains of objectives. These three 
domains are, of course, cognitive, affective and psychomotor. The breadth of this literature makes it 
virtually impossible to review and analyze it completely. As an illustration of the variation across 
disciplines, a survey of those disciplines incorporated into Crookall, Greenblat, Coote, Klabbers, and 
Watson (1987) include language and communication, social issues. 3nd management and business In 
other literature, instructional simulation includes the areas of medicine, health, aeronautics, library 
and information science, military science, and political science. 
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In our next section we share a brief overview of instructional methods. In this introduction we 
have argued that issues about instructional simulation are best implanted within comparisons and 
contrasts with other methods. Again and again in this review we will haunt this argument. The 
following brief section is an overview wliich should provide the framework for comparisons. 



Instructional Methods 

In this section we present an overview of instructional methods. We have not attempted to 
discuss these in any detail. This list and a few associated issues will abet our discussions of 
similarities and differences between simulation and other methods. No single instructional method, 
not even instructional simulation, should be treated in isolation from other methods, nor should any 
be revered as necessarily "innovative" while all others are relegated to "traditional," an 
argumentative device so often observed in instructional literature. The strengths and weaknesses of 
instructional simulation should not be compared to all other methods together, as do some (see 
Sharrock and Watson, 1987, for example). Simulation should rather, be compared to other specific 
methods in specific contexts. Figure 1 is a list of other instructional methods to which comparisons 
will be made during this review. 



Figure 1 Instructional Methods 



Lecture 

Independent reading 
Demonstration 



Lecture-discussion 

Independent study (reading plus overt responding) 
One-on-one tutoring or conferences 
Small group tutoring or conferences 
Dialogue (Socratic) 



Cooperative learning (many forms) 



Discovery (should read ''Constructed" learning) 



Case studies 
Simulation 



Field (or clinical) experience (sometimes called direct experience) 
Cognitive apprenticeship 
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The methods listed are not necessarily independent of all others. It seems that the list is in some 
ascending (or descending) order, but the description or labeling of that order is tenuous. The 
classifications such as those which appear in West, Farmer & Wolflf (1991, pps., 253-254) seem 
inadequate. Some reasonable classifications exist, but are weak from the standpoint of mutual 
exclusion. A taxonomy would probably be impossible, but a classification system may be. Some 
possible bases for classifications systems may include: 

Independent vs. guided; 

Individual vs. group; 

Reception vs. construction 

(Discovery and reception are dated metaphors for learning.); 

Teacher managed vs. student managed; 

Passive vs. active 

(Not much potential for this. A reader or listener can be as intellectually active as a 

participant in an apprenticeship. There are possibilities that learners can be physically 

active without being mentally active.); 
By objective or learning goal 

(On the surface this does not have much appeal as a possibility.); 
By level of expertise of student required 

(From novice through advanced to expert.); 
By instructional control. 

Who controls pace? (Tutor as machine or teacher or student?); 
By planning/design (Amount before instruction.); and 
By cost. 

Related Questions 

Considering the emphasis which should be placed on comparing the simulation with other 
instructional methods, it is important for those involved in the simulation to reflect on a number of 
questions. Is instructional simulation mutually exclusive from the other methods listed? With which 
does it overlap? From which may it be separated? We will not pursue these issues on other 
methods of instruction in this paper. We have mentioned them, however, for comparison purposes. 

We now turn to a discussion of the constituents of a good instructional simulation. There are 
several listings In the literature, each of which are valid. 



1 < \ 
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CflngtifajUfflla Of » "flood" Instructi onal Simulation 

(1) What Are It « Fundamental Components? 

There are several listings of fundamental elements of simulation games in the literature. We have 
followed the terminology of the sources of these listings and note here that "games" or "gaming" is 
often used as a synonym for simulation. We enumerate in Figure 2 those found in relatively recent 
literature. From these listings, it is clear that considerable variability exists in the literature about 
the fundamental components of games and/or scenarios. This does not inherently mean that there are 
major disagreements among experts. Given the variation in instructional simulations discussed 
earlier, particularly in the types of games [unregimented play, simple role playing, systems games, 
simulations relying on predetermined roles or action cards, person-computer, reactions simulations, 
and monopoly-type box games (Davis, 1980)1 one should expect that components will differ. 

(2) What Is It Designed to T each (i.e.. Whv Is It Used?)? 

Greenblat (1980) and others (Thiagarajan and Stolovitch, 1978; Adams, 1973; Dorn, 1989; and 
Szafran & Mandolini, 1980) have documented that advocates of simulation have claimed that 
simulation can be used to teach practically all levels of objectives from both the cognitive (fact 
learning to problem solving including what, when, how and why) and affective domains (awareness of 
affect to development of systems of values). Please see West, Farmer, and Wolff, (1991), pps. 46-49; 
Bloom, (1956); and Krathwohl, Bloom, & Masia, (1964) for further discussions of varieties of 
educational objectives. 



ERIC 



11 



Figure 2, Fundamental components of Simulations 



Duke (1980): 

1. Scenario (plot outline); 

2. Pulse (event introduced to 
focus players' attention); 

3. Cycle sequence; 

4. Steps of play (sets of 
instructions); 

5. Rules; 

6. Roles; 

7. Models (systems for 
keeping a record of 
progress through the 
game); 

8. Decision sequences and 
linkages (connections 
among what players are 
doing, when, una how); 

9. Accounting system 
(results of steps of play); 

10. Indicators (characteristics 
of the accounting system 
which are emphasized for 
the players); 

11. Symbiology 
(representations of 
indicators); and 

12. Paraphernalia (materials 
required to run the game). 



Thiagar^jan and Stolovitch 
(1978, p. 15)2 

Critical characteristics 
include: 

1. Conflict; 

2. Constraints (rules); 

3. Closure (ending with 
method for determining 
winning and losing); 

4. Contrivance (artificial 
activities); and 

5. Correspondence (parts of 
game are matched with 
the situation represented). 



Variable characteristics 
include: 

1. Type of conflict; 

2. Type of constraint; 

3. Type of closure; 

4. Degree of contrivance and 
correspondence; 

5. Replayability (Can the 
game be played again by 
players?); 

6. Time requirements; 

7. Equipment and materials; 

8. Number of players; and 

9. Purpose of game. 



Brent (1977): 

1. Roles for participants; 

2. Interactions among roles; 

3. Rules governing 
interactions; 

4. Goals of interactions; and 

6. Criteria for determining 
goal attainment and game 
termination. 



Benson, McMahon, ft 
Sinnreich (1972), parts of 
scenarios: 

1. The character of all 
organizations in the game; 

2. Team values (goals or 
ideals); 

3. Team resources; 

4. Interteam relationships; 

5. Intrateam relationships; 
and 

6. Player roles. 



Simulation advocates have generated numerous claimed outcomes for simulation. Some are as 
follows. Instructional simulation helps: 

1. students learn collaboratively (Thiagarajan and Stolovitch,1978, p. 61); 

2. students have fun (Adams, 1973) (as in "Learning should be fun."); 

3. student motivation, excitement, interest (Adams, 1973, Greenblat, 1988, page 16); 

4. student appreciation of group dynamics (Adams, 1973); 

5. student understanding the structure of knowledge (Adams, 1973); 

6. students as they connect cognitive and affective learning (Adams, 1973); 

7. students make later learning more meaningful (Greenblat, 1980); 

8. students bridge gaps between the conceptual level and real life situations (Horn and 
Cleaves, 1980, p. 58); 

9. by conveying information or facts, holistic impressions (gestalts), and relationships 
(Greenblat, 1988, page 16); 

10. develop skills such as critical thinking, analysis, problem -solving, interactive, and 
communicative (Greenblat, 1988, page 16); 

11. student self evaluation (Greenblat, 1988, page 16); and 

12. students develop attitudes (Greenblat, 1988, page 16). 

We are not declaring that instructional simulation cannot contribute to the attainment of any or 
all of these objectives. Our major point is that most sweeping claims generally are not yet empirically 
based (Greenblai, 1980), an issue which will be discussed in greater detail later in this report. 
Furthermore, given the wide range of simulation types, objectives, and disciplines within which 
instructional simulation has been used, any categorical claim for simulations effects per se is 
unwarranted (DeNike, 1976). It is time to surrender the broad, sweeping claims for effects of 
instructional simulation and become specific and empirical. The time has arrived to investigate 
empirically the planned effect for the specific simulation for the stated purpose, points 
elaborated by Bredemeier and Greenblat (1981). 

As does Dorn (1989, p. 1), we visualize simulation as "one tool on the pedagogical shelf which 
can be used in a variety of ways for many purposes. We, in all fairness to the advocates of 
instructional simulation, recognize that advocation of most instructional methods has been 
characterized by rhetoric and ideology rather that empiricism. This has been true of most of the 
instructional method innovations from discussior to discovery learning to cooperative learning, as 
examples. 

In a later section we will discuss the various studies which constitute evidence for effects of 
simulations on various types of learning. Given the range of simulation across disciplines and 
objectives, no categorically positive claims for effects will likely be warranted. 



(3) Within the Simulation. W iiere/how Does the Required Learning Take Place? 



There can be no specific or definitive answer to this question across all students and all 
instructional simulations. In general, however, learning can take place from the moment an 
instructor an-iounces that a simulation will be used weeks or moments from the use of the simulation, 
or during any phase of the simulation from giving instructions through debriefing. This is similar to 
the discussion by Duke (1980) that each cycle in a game reinforces any knowledge gained in previous 
cycles. 

Guide for Using Instructional Simulation in the Classroom, This leads to an intuitively based 
set of phases for using instructional simulations. While the research on these phases is sketchy and 
imprecise they have some empirical support and considerable experiential support (Lederman,1984) 
and seem intuitively sound. Phases are as follows: 

1. Introducing the game [Provide an overview of learning goals and topics (Dorn, 1989), not 
specific objectives. Include instruction on how the game is played]; 

2. Playing the game [Interrupt (Dorn, 1989) game play if necessary for running or if it 
promotes learning.]; 

3. Debriefing, including discussion (Livingston and Stoil, 1973) of how main points fit into 
earlier instruction (Dorn, 1989) [This should consist of comparisons with prior knowledge 
and with in situ applications. There is even some evidence that a sequence such as 
playing the game then discussion then replaying of the game is advantageous (Kidder and 
Guthrie, (1972)]; 

4. Supplement with other methods (Bredemeier, 1978, Dorn, 1989); 

5. Testing and evaluation (Dorn, 1989) (Formally evaluating the simulation and testing the 
students will increase the probability of effective learning and subsequent efficiency of use); 
and 

6. Reviewing and restating major points of the simulation and integrating those points during 
subsequent instruction when other method are being used. 

These various phases of uses of simulation may involve lecture, discussion, and independent 
reading and could involve cooperative learning. Once again this shows how simulation cannot or 
should not be considered a method completely separate from other methods (Duke & Seidner, 1978, p. 
18), In actual fact, instructional simulation, properly used, is a synthesis of instructional 



methods. Following these phases can increase the probability of learning from simulations, of 
gaining the full benefits. 

(4) What Are the Wavs to Measure Learning? The Rest Wav? 

The measurement of learning from a simulation should not be considered as distinct from the 
measurement of learning from any method listed in Figure 1 (list of methods). If there is a best way 
for all methods, it is placing the learners in the context (in situ) in which the skills, attitudes, or 
appreciations are to be used or displayed and systematically observing the extent to which these 
skills/attitudes or appreciations are used or displayed by those learners. This usually entails delay 
from presumed learning to testing and is extremely perilous in the cases of some performance areas 
and prohibitively expensive in most. Thus it is only best from the stance of validity: validity in this 
use meaning accurate measurement and precise allowable inference. 

In most practical measurement, written, oral, or performance tests can be used shortly after the 
classroom experience whether that experience is a lecture, a discussion, an apprenticeship or a 
simulation. In some instructional simulations, particularly computer simulations, however, records can 
be kept throughout the simulation of the responses of the learner. This response record, based on 
careful programming, can take the place of many traditional testing or learner assessment devices. In 
some instructional simulations this would be the major virtue of computer-based simulation, as 
opposed to some other type of simulation. 

(5) Is There Cost/Benefit over Oth er Teaching/ Learning Methods? 

Given the wide range of types of instructional simulations noted in earlier sections of this paper 
and the large number of other methods listed in Figure 1 there is no reasonably complete answer to 
this question. Rational comparisons may be made, however. Assuming the comparative method is 
equally effective in learning outcomes: 

1. Instructional simulation is typically more expensive than lecture or discussion; 

2. Instructional simulation is, in most disciplines, with most objectives, less expensive than 
clinical or direct experience or cognitive apprenticeship because every student must be 
placed within a job or professional context while a simulation can be run over and over 
again with numerous classes; 

3. The "up-front" costs, such as design (research and development) costs of instructional 
simulation, are typically higher than many of the other methods; 
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4. It is important to consider that often learning in formal classrooms suffers from deadening 
sameness, a lack of change of pace [Simulations can also become boring when continuously 
used (Bower, Bersamin, Fine, Carlson et al (1974)]. No method is categorically better than 
any other method, but simulation can provide a welcome and justifiable change of style, 
approach, or method, even when it is more expensive; and 

5. Simulations are sometimes preferable over clinical or direct experience (Locatis & Atkinson, 
1981) or cognitive apprenticeships. Especially when direct experience (a) endangers learners 
(or others) in the experiential situation (b) is difficult or impossible to provide (c) is too 
complex for the novice to comprehend and (d) unfolds so slowly that valuable 
interconnections are lost. 

Therefore, within a curriculum or program context, day in and day out, hour in and hour out, 
most of the methods should be programmed into the experiences of learners. There is no single most 
efficient method for all students, for all topics, for all objectives, and for all parts of the curriculum, 
during the long hours in which the curriculum or program unfolds. There are situations, however, as 
discussed in items four and five above in which instructional simulations can have a clearly favorable 
costfaenefit ratio. 

How Are "Good" Simulatio ns Designed/Developpri? 

The most accurate and general answer to this question is that good instructional simulations are 
designed in ways that are similar and consistent with the designing of any type of instruction. In 
many senses the design strategies of a good lecture or film strip are quite similar to the design 
strategies of a good instructional simulation. There are several traditionally acceptable models of 
instructional design which may be applied to the design of any instructional format. 

Perhaps the most traditional, most simply stated, and oldest design strategy is that which had its 
origins in the early years of the 20th century in the time and motion studies (Please see West, 
Fanner, and Wolff, 1991, pp. 2-3 for more on this). 



Figure 3 Five Steps in Instructional Design Cycle 

1. Set objectives 

2. Preassess 

3. Plan instruction 

4. Teach or trial 

5. Evaluate/test 
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Following Figure 3 one would (1) set the objectives of the simulation; (2) preassess the prior 
knowledge, skills, or attitudes of the target learners; (3) plan the simulation; (4) conduct trial runs of 
the simulation to "debug" it; and (5) conduct tests of learning and evaluations of its effectiveness. 

One of the frequently omitted steps in the design of instructional simulation is sufficient trial 
runs. In our opinion, an instructional simulation, particularly if it were to be marketed, should be 
subjected to developmental research similar to a standardized test of intelligence or achievement 
Practically and minimally, several test trials ought to be conducted [as Stolovitch (1976) recommends] 
with samples of the intended learners before the instructional simulation is considered to be 
adequately designed or developed. The evaluation and monitoring of these trial runs should be 
characterized by empirical approaches, dispassionate observers, and, if possible, external review 
(evaluators). Testing and evaluation of these trial runs may result in resetting of objectives and 
replanning, or redrafting the simulation. 

A more modern and complex version of design steps, and one consistent with the psychology of 
instruction and learning until approximately 1970, is that of Dick and Carey (1985, pp. 2-3) which is 
also recommended by Gagne, Briggs, & Wager (1988, p. 22). Unfortunately, this design model, or 
system, has not sufficiently incorporated innovations from the cognitive perspective which has guided 
experimentation in learning and instruction since the early 1970's. 

A new design model, or system, (please see Figure 4) which is even more complex and which does 
incorporate this new perspective and research and development is the model of West, Farmer, and 
Wolff (1991, pp. 209-263). This recent model provides guidance for the designer from the analysis of 
social, economic, and political forces acting on instructional designers (Phase 1) through the 
establishment of aims and objectives, content considerations, learning (or cognitive) strategy 
considerations, and instructional methods and media considerations to the final stage of testing and 
evaluation (Phase 6). Each phase has several steps. 
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Figure 4 Simplified Version of the Instructional Design Template 



1. Situational audit 

2. Aims and Objectives 

3. Contents and Uses 

4. Cognitive strategies 

5. Means of instruction 

6. Evaluation and Testing 
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(1) What Is thfi Fundament al "Process"? 

We have discussed the value of thinking of the design of an instructional simulation as an 
instructional design problem. In our view the failure of instructional simulation designers to follow 
the general instructional design strategies may be due to their taking cues from recreational game 
development, in which the general purposes are entertainment, or operations research, in which the 
general purposes are debugging of systems entailing problem solving. Now we turn to some of the 
more specific design strategies in the instructional simulation literature. First we discuss the issue 
of the extent to which a simulation should reflect the event or situation being represented. This is 
the issue of fidelity or verisimilitude. Then we discuss simulation design strategies including 
Greenblat (1987 & 1988), Thiagarajan and Stolovitch (1978) and Duke O980a, 1980b) and recommend 
some steps in design which could supplement those strategies. 

Fidelity. Metaphor. Modeling and Verisimilitude 

A basic issue in simulation, and one given considerable attention in the literature, is the fit or 
match between the simulation and reality. Unfortunately, this phrase "the fit with reality," or, any 
approximation of it, tosses us into unresolvable philosophical issues. Unfortunate also is that this 
phrase, "fit with reality," is based on naive assumptions: that there is a reality and that humans have 
the cognitive means to know and represent it in some absolute sense. This is the familiar "naive 
realism." Another naive assumption is that correct procedures allow the development of an accurate 
model. The accent is on the singular correct model. Thus there is assumed in "naive realism" one 
reality to know, and one correct way of knowing it. 

Naive realism is generally considered a dated philosophy, a dated model of "reality" actually. 
More recent "models" or perspectives include relativism and phenomenology. Both modern options 
promote visions of multiple "realities" which are socially and personally constructed rather than 
discovered through human observations and interactions among humans and the universe. These 
perspectives or philosophies dominate the natural and social sciences today and have yielded exciting 
applications. 

The ways of knowing attendant with naive realism (its "cognitive" psychology) is essentially that 
what we see, hear, touch, taste is the reality. Events in the world occur and the eye-mind 
informs us. Knowledge is determined by an external reality. On the other hand, the more recent 
cognitive psychology is that the nature of mind and eye, including the contents and processes of 
mind-eye, are more profound determinants of knowing than the event or reality. Or, to paraphrase 
Easterly (1978, p. 24), who places this in the context of simulation, information drawn from the 
environment does not constitute reality, but the designer's perceptions of reality. 
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To paraphrase this more recent view of cognitive psychology (and philosophy), humans probably 
construct their own individual "reality". Just as the astrophysicist constructs models of the universe 
and tests the current model as best it can be tested, other humans create models of how their 
businesses, interpersonal relationships, and governments operate; then they live by/in, and thus test, 
those models. The most current name for those models in current cognitive psychology is 
"schemata." These personal models or schemata largely determine what we sense and what 
we subsequently know. 

All of this may seem far from the problems of instructional simulation; but, once one attempts the 
creation of a model to represent an event (or situation), it is one thing to assume a sole correct 
model. It is quite another to assume that mrny helpful models may be possible each of which are 
joint functions of the situation and the people involved. 

This does not mean that one can or should create models without deep knowledge of the event or 
the knowledge about which the simulation is to be developed. None would recommend the 
development of an instructional simulation, or any form of instruction for that matter, without 
expertise in the particular area, or that part of the discipline the game supposedly represents. It is 
clear that Greenblat (1988) would not, for she eloquently defends deep knowledge of the discipline, 
events, issues, or problem, being simulated. 

Returning to the issue of validity of representation and simulation in the context of modern 
cognitive psychology and philosophy, what are the interfaces between the simulation and that to be 
represented? The task is traditionally and deceptively simply stated: to construct a model. To be 
instructive, most models (or simulations) have to be in some measure like something or some event, 
but like what thing or what evpnt. or what nart of the event ? 

Since we have argued that all knowledge is model (and schema) based, it seems candid to admit 
that when a model is buUt to represent something, then that model is a model of a model. In other 
words, a simulation is unlikely to be a model of a "reality," but it is more likely to be a model of 
some person or persons models or schema. 

From the stance of modern philosophy and modern cognitive psychology let us, as simply as 
possible, outline a more modern statement of the related issues involved in the determination of 
accuracy of the representation with the accompanying task of designing a game for the purposes of 
instruction. 
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Tracing simulation construction through the conventional (and largely correct) procedures as 
outlined by Greenblat (1988, pps. 27 ff.), one 



It is in the first three stages that corrections of statements and, in some cases, reorientations of 
designer activities are needed; because it is here that most developers speak of "the real-world 
system" (for example, Greenblat, 1988, p. 28). It is in these stages that the issue of verisimilitude or 
accurate representation is paramount. The first correction, a verbal statement correction [Please see 
Bruner (1986) and West (1987) for discussions of the centrality of "utterance correction" (verbal 
representation) in education and science], is not to speak of modeling "real-world systems" or "the 
system," but of modeling (simulating) a model- a choice of models among several models which 
probably exist. [Please see Anderson (1987) for further discussions of some implications of the "reality 
problem" for simulation]. 

To some in cognitive science this is tantamount to the creation of a metaphor. The simulation is 
a metaphor, analogy or model of "the real-world system." 

Rather than speak or write of " reality " or the " real-world system " or " the system ", let us speak of 
in situ (the place or situation where people perform and learn and from which simulators attempt to 
develop a model for instructional purposes). For example, in situ for management could be any 
corporation, organization, or business. Designing instructional simulation representing management 
procedures would likely involve observing managers on site and creating a model or several models of 
several observed managerial actions. The observations would likely vary among the sites, if there 
were more than one, and among individuals, if there were several. For simulation development a 
designer approximates a model, but that model is constructed from the individuals observed and their 
models in their situations. 

Still it is essential that the game designer learn a great deal about in situ, the place in which 
models are used and from which instructive models are derived. For example, a designer seeking a 
model for a management simulation should become familiar with management practices in a variety of 
organizations. But there should be acute awareness that several equally operative models may be 
used by individuals in situ. Then an odd thing happens on the way to simulation design: the 
designer's models (schemata) are invested in the reconstruction of the in situ individuals 1 model or 
models. The goal is to simulate, to model, but existing in situ are simulation designers 1 models, 
observed individuals' models, and quite possibly communal (culture- or group-based or 1 



1. 
2. 
3. 
4. 
5. 



Sets objectives and parameters, 
Develops a model, 

Makes decisions about representation, 

Constructs and modifies the gaming-simulation, and 

Prepares the simulation for use by others. 



*1 i Xt 



21 



organizationally -based) models. What is meant by "communal" is the shared conventions of "the way 
things work in the represented field." 

In the communal models, we may become embedded in discipline or in situ variables which are so 
powerful that the procedures of simulation development must be readapted. One very specific design 
implication is to be alert for such differences, to be alert to the communal or shared models of what 
works, especially when what worke is either extremely controversial, not the best, or inadequate, but 
is merely habitual. In these cases the designer's simulation or model should be altered to reflect a 
better, improved method which the simulation will then help to teach. It is known (West, 1981) that 
group-based perceptions, judgments, attitudes and schemata have as profound effects as 
individual-based schemata. 

Knowledge of the relevant discipline is convincingly recommended by Greenblat (1988) and is very 
necessary in designing all high quality instruction. This typically involves studying models of the 
experts in the relevant discipline, particularly in the cases in which one studies documents including 
recorded procedures. Someone or some group devised the documents. Specific questions such as the 
following should be attended carefully. 

o Who devised this material? 

o What is the implicit model? (Models may be as invisible as assumptions.) 
o Is the model a consensus model? 

o Are the informants consistent with consensus in the field? 
o Do informants agree among themselves? 

o If the model is a communal model, is there danger of group-think (Janis, 1972; West, 
1981)? 

o If the game developer's procedure is to observe the system (in situ) directly, what is the 
implicit model of the developer? 

o If the system is a consensus based system, is it troublesome? That is, is there some sense 
of change imminent? Are there controversial issues abounding? Are vested interests 
gaining excessively from the consensus? Has the system become unnecessarily rigid 7 



Figure 5 Competing Models 



Simulation Designer's Models 
Observed Individual's Models 
Other Communal Models 
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It is odd that deep study of the discipline (in situ in our words) receives such emphasis in the 
instructional simulation and simulation literature (please see Greenblat, 1988, pps. 27 ff. for an 
example) given that few educators would advocate any systematic design of instruction using any 
method listed in Figure 1 without knowing the field. Imagine developing a lecture, for example 
without knowledge of the subject. Yet, if simulations designers feel that it is necessary to emphasize 
this, it once again demoi. t**ates the extent of simulation designers needs as well as their general and 
very undesirable isolation from instructional design. All methods of instruction require 
knowledge of the content during their development. 

As we have mentioned, the variables in the knowledge domain, or discipline, can be significant. 
In particular, these questions are critical: Is the knowledge domain ill- or highly-structured? Is it 
overstructured? If ill structured be cautious. By ill- and highly- structured we mean the extent to 
which the field is governed by the rules of logic. Generally there are but two disciplines which are 
highly structured, logic and math. All other fields are generally ill structured and are rife with 
assorted "truths' 1 and procedures. By "overstructured" we mean that there is high social agreement or 
consensus about "truth" or proper procedures when, in fact, there are several "truths" or effective 
procedures, but only one has become acceptable to the exclusion of perhaps other acceptable "truths' 
and procedures - to the point of becoming excessively rigid. 

There are at least two types of systems which may be represented in simulation, thus two types 
of models: personally constructed (In which case go to the expert!) and consensually, socially 
constructed (In which case, go to the experts collectively!). Rorty's (1979) truth is "what works," and 
we add "what is thought currently to work." 

Conditions Which Determine the Instructional Value of Verisimilitude or Fidelity 

Verisimilitude, or, as it is in some texts referred to as fidelity, is generally viewed as an important 
and desirable characteristic in a simulation. This may be a transposition from simulations in 
operations research to instruction. In operations research fidelity is practically always imperative, In 
instructional simulation, however, fidelity is not always desirable for a variety of reasons. There seem 
to be three classes of variables which influence the desirability of verisimilitude [matching in situ 
characteristics (particularly complexity) with characteristics of the instructional simulation]: 
L Learner variables 

stress tolerance 

anxiety 

ability 

prior experience 

tolerance for noise in an information system 
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II. Task variables 

complexity [Varying complexity within a simulation may be desirable should simulation 
constraints, rules, and variables become more and more like external models 
(more complex)] 

importance 

rapidity of output required 
noise management requirements 

III. In situ or model variables 

complexity 

speed of running (when unfolding of in situ events occur so slowly that valuable 

interconnections are lost, for example). 

This list helps to underscore the difficulty of deciding on the extent to which verisimilitude should 
be incorporated into an instructional simulation. We are, obviously, arguing that the value of 
verisimilitude *s not trivial (Lederman & Ruben, 1978, Elder, 1973, Ruben & Lederman, 1982, & 
Sharrock and Watson, 1987) and that it is an interactive function of at least three types of variables: 
learner, task, and in situ. Each of these types of variables potentially subsume many more specific 
variables. In the final analysis, empirical methods should be deployed to determine the value of 
verisimilitude, particularly in designing simulations for highly sensitive, dangerous, or extremely 
expensive instruction. 

Levin and Waugh (1988) have suggested an interesting and potentially powerful way of managing 
complexity and fidelity (or verisimilitude) in simulations for learners. Their idea is to provide learners 
with a range of fidelities in which the novice begins with simple settings and moves toward more 
complex settings. They use the phrase dynamic support for decreasing the amount of support 
necessary for novice learners as they engage increasingly complex simulation settings. 

A major consideration about verisimilitude is to design the simulation to "bring the learner along," 
to cause gradual learning. It may often be necessary to gradually increase complexity when 
performance in situ is complex, or the situation is complex, otherwise the learner may be 
overwhelmed. Too great a match between the game and the complexity of efficient in situ 
performance can often overwhelm the learner, particularly at novice levels of learning. For example, 
for flight training simulation should usually have low fidelity initially, with gradual introduction of 
greater fidelity. 

Another important consideration involving fidelity is credibility (Elder, 1973), the extent to which 
the learner believes that there is matching between the simulation and in situ. The credibility of the 
simulation increases the probability of learning, up to a point. Fidelity increases credibility, generally. 
Too much fidelity decreases the probability of learning complex tasks. Thus some instructional 



simulations become caught in crosscurrents of learning, credibility, fidelity, and complexity. 
Consideration of the three types of variables listed above can help designers instructional^ manage 
those crosscurrents. 

In the next section we discuss further simulation design issues. We begin with a presentation of 
the more widely used and cited systems. 

(2) What Are the Discrete Design St eps of the P rocess? Are there Shortcuts? Alternatives? 

We have located several design systems which seem to predominate in the field of instructional 
simulation. In the following pages we list the discrete steps and, later, attempt a comparison and 
contrast. Later we offer some additional or supplementary design procedures. 

Of the first two systems, the Thiagarsgan & Stolovitch (1978) is the more detailed. Notice that all 
three procedures prematurely stop once it is found that the system can be made to "run/ 1 It seems 
to be accepted that if the game can be played by the rules without the developer being present, then 
it is a success- and that instructional objectives were attained. It is intriguing that Duke's (1980) 
procedure for step eight is that the client conduct the evaluation. There are no steps or phases 
included for the testing of learning or the evaluation of the success in terms of the effects of the 
game on learners, unless, in the procedures of Duke, the effects are part of the criteria specified in 
step 1. To us this lack is a recurring sign of the isolation of gaming design from instructional design 
and instructional research. As we have said, this may be a result of instructional simulation design 
being more influenced by the design of simulations for entertainment or for operations research and 
development than it is influenced by instructional design. 



Greenblat (1976, 1988, ppt. 27 ff.)i 
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1. Set objectives and parameters. 

2. Develop model. 

3. Make decisions about representation. 

4. Construct and modify 
gaming simulation. 

5. Prepare simulation for use by others. 



or,; 



Figure 6 Simulation Design Systems 



Thiagarajan A Stolovitch (1978), 
checklist of steps, p. 68 

1. Define the instructional topic. 

2. Construct a model to reflect the real-life 
event. 

3. Select a suitable game format. 

4. Identify the major characters, resources, 
and constraints. 

5. Specify the overall game sequence. 

6. Specify the termination rule. 

7. Establish the criteria for winning. 

8. Design a sequence for each round. 

9. Write a background scenario and role 
descriptions. 

10. Assemble prototype materials and 
equipment. 

11. Teat the game with players and revise. 

12. Write the player's manual. 

13. Test the game under "hands-ofT 
conditions and revise. 

14. Specify the outcomes of the game. 

15. Prepare the administrator's manual 



Nine basic steps of Duke (1980a, 
1980b): 



1. Develop written specifications for game 
design; 

2. Develop a comprehensive schematic 
representation of the problem; 

3. Select components of the problem to be 
gamed; 

4. Plan the game with the Systems 
Component/Gaming Element Matrix 
(column headings - game roles, row 
headings - steps of play); 

5. Describe the content of each cell in the 
matrix (step 4); 

6. Search "repertoire of games" for ideas 
for each cell in the matrix 

(step 4); 

7. Build the game; 

8. Evaluate the game using the criteria 
of step 1; and 

9. Test the game in the field, and modify. 
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It is unlikely that there are any shortcuts to good design. All improvements of instructional 
design practices are intimations of making design more, rather that less, complex. If instructional 
design simulation is to be brought into focus within instructional design and development in general , 
and if instructional simulation is to fulfill its potential, more complex strategies are probably 
warranted. There should be more designer viewing of the on site situation (e. g. task analysis) or 
more study of the discipline. There should also be more designer produced models of in situ 
processes and practices, particularly in ill-defined domains and situations (up-front simulation design 
work), as well as more testing and evaluation efforts to observe the short- and long- term effects of 
the specific instructional simulation. These are better approximations to both the effective and the 
ideal forms of instructions- whatever the method. 

Procedural explanations such as those design steps outlined in Figure 6 are often overdrawn, 
excessively linearly presented, and not intended to be followed as sequentially as the lists appear, 
Good simulation design will incorporate these steps and numerous substeps or subroutines, some 
mentioned and some not, but will not often be as orderly (Easterly, 1978) or as linear as represented 
above. 

At any point in the procedures enumerated in any of the systems in Figure 6 the designer may 
need to return to early steps or forecast shapes of steps not yet embodied or, even firmly conceived. 
Good technological guidelines exist; but they are, after all, guidelines, not immutable laws. This is 
apparent from the variances in the design steps enumerated in Figure 6 and from the interviews 
conducted. 

When one designs instructional simulations shortcuts may be taken, generally, only at the peril of 
learning. What appears to be shortcuts usually are understatements or oversimplifications of 
procedures. While one expert (Greenblat, 1988) has listed fewer steps than others (Thiagarajan & 
Stolovitch, 1978) the difference is, in our opinion, more in Greenblat's preference not to list 
subroutines than in real differences of procedures. 

Improvements in Design. Any of the three systems included in Figure 6 obviously work. There are, 
however, suggestions we can make which can, if followed, improve any of the systems. First, a task 
analysis (Jonassen, Hannum, & Tessmer, 1989) or its cognitive (Braune & Foshay, 1983) or affective 
counterparts may be needed for the setting of objectives or defining the topic of the simulation. Task 
analysis is basic to systematic instructional design. One caution, however: task analysis can aid the 
uncritical recording of current in situ practice and serve to only reinforce current, and flawed practice 
if fed into the instructional simulation. Any model or simulation can become seductive (Please see 
West, Farmer, and Wolff, 1991 for further discussion of seduction in instruction, particularly when 
metaphors are used and simulations are metaphors), that is, the simulation is erroneously taken to 
perfectly match in situ. When current in situ practice is known to be flawed, inefficient* or extremely 
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controversial, task analysis may primarily serve to perpetuate those flaws, inefficiencies, or 
controversies. 

Our second design recommendation is to provide for systematic psychometric evaluation of learning 
from instructional simulation. As a symptom of empirical, psychometric neglect, note the scarcity of it 
in the recent 342 page proceedings of the 17th International Conference of the International 
Simulation and Gaming Association (Crookall, Greenblat, Coote, Klabbers, & Watson, 1987), the 212 
page volume edited by Hoilinshead & Yorke, (1981), the 216 page volume of Ellington, Addinall, & 
Percival (1981), the 168 page volume edited by Megarry (1977) and the 375 page volume of Shears 
and Bower, (1974) . It is very likely that this inclusion in design would help to dispel many of the 
broader and unsubstantiated claims for effects of instructional simulation. It is ateo likely that it 
would result in more instructionally powerful simulations. 

(3) How Are the Objectives for the Simulation Est ablished? 

In this there is no substitute for entering the situation in which the skills to be taught are to be 
used or applied and conducting a task analysis , or its cognitive or affective equivalent, as we have 
emphasized earlier in this report. The idea of an equivalent is significant because so many 
simulations are not intended to teach performance of "tasks" per se; but, rather, interests, attitudes, 
appreciations, and higher level intellectual operations. The basic idea is to find out what 
performance, intellectual operation, or attitude is deemed adequate or appropriate in situ (the place in 
which the skills are to be deployed) and describe it in detail. 

One example of what we mean here is the designing an in-basket simulation, for example, to 
teach federal acquisitions. In other words the designer of this in-basket simulation should conduct an 
analysis of federal acquisitions in situ to determine how it is done. If the intention is to teach 
attitudes, then what are the attitudes present? If the intention is to teach the intellectual skills or 
information about procedures, then what are the procedures in place? In many cases the instructional 
objectives may be transcribed from the site by conventional task analysis. 

But as we have discussed in an earlier section, if the in situ procedures are considered to not be 
working properly in situ or if they are controversial (e.g. models or schemas ol experts are found to 
differ) then the direct transcriptions of skills to objectives may be very problematical. The "task 
analysis" may have to be supplemented or even replaced by defaults such as reliance on informants 
or experts. The choice of informants or experts then becomes a problem. 

In summary, however, the typical strategy is to conduct an analogue of the task analysis and 
transcribe directly to objectives. It should be emphasized that there is little reason to be preoccupied 
with stating them in behavioral terms (Locatis & Atkinson, 1981 and West, Fanner, & Wolff, 1991). 
In most cases this transcription works. Care should be taken not to myopically focus on behavior. 
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This is generally not a problem in the simulation literature and practice. If, however, the objectives 
are behavioral in nature, it may be desirable to determine the affect and the intellectual operations 
associated with the behavior. This focus, of course, depends on the intended outcomes and other 
variables. 



(4) How Are the Objectives Translat ed into (Incorporated in) the Learning Vehicle? 



Once a task, affective, or cognitive analysis has been conducted, the most conventional strategy of 
translation is to take each detailed step in the analysis and turn it into an objective or a series of 
objectives. If each step is to be incorporated into the simulation, a miniature representation of each 
step, or phase of the analysis must be conceived by the designer and incorporated into the rules, 
routines, cycles, scenarios, components, criteria, or constraints of the games. 

We recommend a systematic checklist for this operation consisting of the following: step or part of 
the analysis; corresponding objective of game; and corresponding component of game. Please see 
Figure 7 for a recommended checklist. If each part of the task, affective, or cognitive analysis is 
important then corresponding objectives should probably be conceived and specific components of 
games ought to be designed for each. Such a checklist should provide help for this incorporation. 



Studies of Effects and Implications for Evaluation 



In this section we will present an overview of literature on the evidence that simulations effect 
learning as well as how to internally evaluate an instructional simulation by the analysis of its 
contents. Since "let the buyer beware" is especially important in simulation, the latter can be very 
important to a user who may not have the desire or the resources to evaluate a simulation any other 
way. 
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Figure 7. Checklist for Incorporating Objectives 
into the Instructional Simulation: 
From Task Analysis to Game Component 
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(5) How Is the Achievement of these Objectiv es Demonstrated? 

As we have stated in an earlier section above, in well established instructional simulation design 
systems, the typical case is that many simulation designers assume that the conceptualized 
instructional objectives will be achieved if the simulation "runs" without bugs. This is equivalent to 
believing that if a classroom discussion is held, or a lecture given, the stated objectives will be 
achieved. Pedagogically this is unsound and naive. No instructional method has been found to be so 
robust a treatment even occasionally, and certainly not routinely. 

The demonstration of the extent to which instructional objectives have been achieved should be a 
matter of systematic empirical observation, us ually after the instruction. There is a rich and 
practical testing and evaluation research and development literature which provides a wide range of 
testing and evaluation techniques. There should be evaluative observations which are focussed on 
each important learning goal of the simulation. 

While this evaluation of achievement is most often conducted after the instruction, it is possible to 
design some instructional simulations, particularly if they are computer-based so that the assessment 
of learning indicators are conducted systematically and recorded and monitored during the instruction 
itself. This is not restricted to testing low-level knowledge but can be extended, with careful and 
systematic design, to higher level cognitive and assorted affective goals. Responses students make 
and the patterns of those responses can be evidence of assorted kinds of cognitive and, properly and 
carefully sequenced, affective change. What that evidence constitutes is largely a matter of careful 
design, programming, and inference. 

Even when testing and evaluation are conducted during the "run" of a simulation they should not 
be taken as the best evidence for learning (attainment of objectives). For most instructional objectives 
the demand is for long-term changes in behavior, cognition, or affect. Educational institutions invest 
heavily in the expectation of this long-term change. The best, then, as we have argued in other 
sections of this report, is to observe the former students performing in situ- that is, in a working 
environment-- after the instruction. 

In the following paragraphs we will discuss and present (Please see Figure 8) some of the 
attempts to empirically evaluate learning from simulation games and the existing reviews of attempts 
to research effects. Categorically, instructional effects for simulations cannot be claimed for a variety 
of reasons including student ability (Thorpe, 1971), variations in use patterns by the instructor 
(McKenny & Dill, 1966), and the tremendous range in quality and type of simulations. 
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In Figure 8 we will summarize several studies and reviews of effects of simulation on various 
dependent variables such as assorted types of learning, attitude change, and motivation. Column 
headings of Figure 8 are study (or review), specific simulation or topic if applicable, objectives, and 
findings. In the findings column in the cases of reviews, the first number represents the number of 
studies said by the reviewer to have results in which there were differences, or effects favoring 
instructional simulation. These reported effects are not always statistically significant. For some the 
appropriate phrases are "in the direction of 1 and "tends to support" which many social scientists would 
reject as merely not statistically significant. 

One of the more thoughtful and carefully done early reviews is that of Pierfy (1977). He divides 
evaluation research on instructional simulation into three types. The first is descriptive studies of the 
effects of a game on one group of subjects. The second is explanatory in which the researcher tries to 
establish that the game had an influence on particular subjects. The third type of evaluation is the 
comparison of learning from simulation with learning from another method. Despite the favorable 
ratio for simulation noted in Figure 8, Pierfy has little faith in the validity of the studies themselves. 
Among the problems he discusses with the research are lack of representative subject samples, 
experimenter bias, failure to control for critical variables such as instructor differences, and validity 
(accurate measurement) and reliability problems with the tests or instruments used for the measures. 

In another thoughtful review Bredemeier and Greenblat (1981) emphasize the disparity of the 
impressionistic evaluation versus the psychometric evaluation. They also emphasis the difficulty of 
determining the effects of instructional simulations. Some of the variables which they think mediate 
the possible effects of simulations are: administrative variables, such as instructor characteristics, 
nature and quality of introductions and debriefings, and variations in game play of the same game; 
internal game variables, such as fidelity, quality of the design strategy, and the matching of game 
characteristics with intended outcomes; group variables, such as group size (Gentry, 1980), and group 
dynamics; and student characteristics such as attitudes toward the game prior to playing the game, 
sociometric grouping (Brand, 1980), sex of players, cognitive style (DeNike, 1976), personality, 
academic ability and game ability. In the Bredemeier and Greenblat (1981) article, the basis for the 
claims of the effects of these mediating variables differs. Still, it is very likely that many of the 
dissonances in the literature on effects of games are attributable to these and other variables. 



Figure 8, Overview of Reviews and Single Studies of 
Psychometrically Observed Effects 
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In a more recent review, Dorn (1989) located numerous references which portray the complex and 
conflicting picture of simulation evaluation research. This more recent portrait generally is not 
substantially different from findings of earlier reviewers. 

It is unclear whether or not instructional simulation is particularly superior to any other method 
for any given objective. There are not enough properly designed experiments to allow this inference, 
While much of the literature reflects a belief that simulation is superior as a method for such 
objectives as increasing motivation to learn, interest in the subject, or affective change, the 
psychometric evidence is very mixed. Thus, educators have every reason to be wary about any 
claims of categorical superiority over any other methods. 

On a more positive bent, while the portrait of effects is indistinct, researchers have provided 
sufficient evidence that instructional simulation serves as, in Dorn's (1989) words, "one more tool on 
the pedagogical shelf. 11 Looked at from this perspective, one selects a method for a purpose within 
a course or curriculum, not a method for a course or curriculum. It is clear that a properly 
designed simulation, properly used, can be a powerful instructional method. It is not at all 
clear that simulation is categorically superior to any other method. Nor should there be 
any assumption that any type of specific objective can necessarily be better achieved with 
a simulation. 



(6) How Do the Designers Knnw when Thev Are 'There?' 

There is no adequate substitute for objective, dispassionate, empirical testing and evaluation, 
After running the simulation with adequate samples of persons, after conducting the debriefing, after 
everything to be done with the simulation has been done, do adequate samples of learners exhibit the 
behavior, or affect, or cognitive process that was the objective and, to what extent? This is usually 
an adequate way of evaluating an instructional evaluation. Even better is to wait until students are 
in a work situation and see if they exhibit the intended outcomes of the simulation. Effective 
instruction whether it is a lecture or a simulation- is that from which the learner exits having 
learned what was intended and that which persists to be used in situ. 

There is, however, another - and, at best, complementary- approach to evaluating an 
instructional simulation game. That is to conduct a content analysis to determine if the game has 
characteristics deemed desirable. This complementary approach is crudely similar to art criticism, 
whereas the empirical evaluation by effect on the learner is similar to engineering. These content 
analyses range from simple and descriptive judgments including such considerations as topic of game, 
number of participants, run time, and so on (Ruben, 1980) to the more evaluative including such 
considerations as significance, validity, flexibility, accessibility, and so on (please see Ruben, 1980 for 
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more detail of these considerations). In Figure 9 we include several criteria (derived from Horn and 
Cleaves, 1980) in a frame which could be used as a guide to conduct an evaluation by content 
analysis and a checklist for recording results. It seems to be consistent with recent views among the 
simulation game developers and evaluators. The criteria included in Figure 9 vary from merely 
descriptive to more evaluative. 

Implications for Users. Those who plan to use instructional simulations developed by others should 
be mindful of "let the buyer beware." Typically designers do not systematically, or psychometrically 
evaluate their simulation for learning effects. Until users demand this, and become more 
sophisticated consumers, users must either first purchase a simulation and later collect their own 
learning effects data or merely assume that the simulation is as cost effective as any other 
instructional method. Our view is that every choice of a method suffers from this lack of 
demonstration of learning effects: there is no best method of instruction. Instructors and 
educators must learn to evaluate empirically what is done in the classroom. They must 
learn to try a method and test for effects, and, perhaps, try another and test it for effects. 
When costs are higher with any method, the necessity to demonstrate effects is greater. 

Conducting Local Evaluations. To accomplish this demonstration of effect we recommend that 
users follow a procedure similar to the following. First, carefully attend to the learning goals or 
objectives o* the simulation used. Second, create a pretest consistent with those objectives or goals. 
Third, administer that pretest to students. Fourth, run the simulation on the class. If possible and 
feasible, have other classes as controls. For example, if there are two sections of the same course, 
withhold the simulation from one class which was selected randomly. Keep all classes as similar as 
possible except for the simulation treatment. Fifth, administer a posttest. Sixth, examine the results 
for significant or important differences on test performance. This is not a perfect evaluation plan 
but it is feasible in many situations. 
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Figure 9 
Simulation Evaluation Frame 
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Figure 9 continued 
Simulation Evaluation Frame 
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(7) How Do the Students Know when Thev Are 'There?' 

This is the best: when it can be graphically demonstrated to the students that they have learned 
what was intended and/or some valued incidental learning. As we stated earlier, this ideally happens 
after the learner takes skills to the site of use. More realistically, the tried and true design strategy 
of pretesting operates well, but at this point it operates to provide the demonstration of change from 
skills mastered or not mastered prior to the simulation to skills mastered or not mastered after the 
simulation. But, again, the best demonstration is well after the simulation when the learner is on the 
site of the intended use of the skill This is too late for the learner who has left a program of 
instruction, but it provides valuable information for those yet to enter the program. 

In so many formal learning situations the instruction is evaluated solely by the opinions of the 
students, that is, whether or not they feel that they learned. This is generally the weakest form of 
instructional evaluation, but its virtue is that it is easily conducted. 

Conclusions 

There is excessive reliance in the literature on the dichotomy "us" (the simulation designers and 
advocates) us. "them" (those who use any or all other instructional methods). This is a familiar 
dichotomy used by so many entrenched in the self-assigned and self-serving role of innovator. This is 
very typical for advocates of any instructional method. It is true of those who advocate discussion 
versus lecture and it is true for those who advocate cooperative versus competitive methods. 

There are excessive, unnecessary, and nonempirically based claims of outcomes for simulation. 
While we have found evidence that some simulations produce claimed desirable effects, many claims 
are made which are unwarranted. This should not, however, substantially detract from the 
willingness to use instructional simulation, to try it, and to evaluate its learning effects locally using 
the step by step plan we recommended in this report or one similar to it. 

Instructional simulation is a worthwhile instructional method which has had some proven positive 
outcomes. There are conditions in which it is cost effective and we have enumerated those conditions. 
The probability of an instructional simulation being worthwhile should be increased if the design and 
evaluation recommendations made herein are followed. 

There is an unusual frequency of reliance on claimed outcomes that are affective in character, and 
thus assumed naively by advocates to be practically impossible to measure. It is naive to assume that 
affective learning cannot be empirically assessed. The psychometrics of affective measurement 
continue to develop. 
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In this report we have responded primarily to three categories of issues or questions (components 
of good simulations, proper design, and evaluation and testing for effects) about current practice and 
have argued that current practice is quite good, with a few exceptions such as the failures of 
simulation designers (1) to conduct task, cognitive, or affective analyses during early phases of design; 
and (2) to psychometrically evaluate the instructional effects of simulations prior to marketing the 
simulation and (3) to include that as a part of design. In the absence of the latter we have 
recommended that users conduct empirically based evaluations of simulations themselves. We 
included a step by step procedure of accomplishing this. We have also recommended that they use 
the detailed checklist provided to conduct a content analysis of simulations offered to them for 
purchase. We have also presented a step by step guide for the proper instructional use of 
instructional simulation by instructors and educators. 

We think that the future of instructional simulation design and use during instruction will be 
substantially congruent with our recommendations. While research incorporating our 
recommendations would require complex designs, it is not beyond feasibility. Without complex 
research and development designs, instructional method questions and controversies will continue to 
be resolved within rhetoric and bias rather than systematic observation. 
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Computers in Instructional Simulation 1 



James E. Snellen 
Steven L. Murray 



PERSPECTIVE 



A computer does not make possible, in instructional simulation, anything which could not also be 
done by human facilitators. For certain types of educational simulation, the task of execution becomes 
easier, or even feasible, by the addition of computer support. What computer support encourages, and 
in some cases absolutely demands, is a strict degree of rigor and consistency in defining and 
executing: 

The underlying model on which the simulation is based. 

If an algorithmic model (i.e. a model in which all rules and methods are explicitly defined) 
for the simulation cannot be established, then it is not possible to base the simulation upon 
computer driven support. 

The way in which the learner(s) will interact with that model. 

For computer mediated simulations, interaction with the model requires interaction with 
the computer. For putting information into the computer, the options available include 
devices such as the keyset and pointing devices (e. g. mouse, touch panel, light pen, etc). 
Voice input is possible although its development is still at a primitive stage as of this 
writing. For receiving information from the computer, there is the computer screen 
(monitor), the computer speaker (for sound effects) and paper prints. The issue of how to 
facilitate human-computer interaction has been a major subject of study in itself 
(Shneiderman 1986). 

The second consideration is one of interaction style. Input from the learner can range from 
free-form text input to pre-defined menus to video game formats. This range of styles 
roughly corresponds to a spectrum of maximal to minimal cognitive processing and minimal 
to maximal reflexive processing by the learner. 



r er to di8CU " the role of computtri In instructional simulation, it ii necessary to use tome of the technical jargon from 
the acid of Computer Science. At an aid to the reader, this paper includes a glossary which defines those terms which may 
otherwise be obscure. Because this paper deals with the Application of computer technology to instructional simulation we have 
attempted to define the terms in the glossary in a way that is meaningful in the context of computer bated instructional 
simulation. Alto we have simplified (In tome respects, oversimplified) some of the concepts in order to make it easier to 
understand the general concept of the terms. 
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The way in which evaluation data will be collected during the execution of the 
simulation exercise. 

The question to be considered here is whether or not we want the learner to be directly or 
indirectly involved in the evaluation. If the learner is to be directly involved then the 
possible means of evaluation available include pre/post testing and self-evaluations. If the 
learner is to be indirectly involved then transactional recording and analysis is an obvious 
choice. 

One great advantage of using computer support for simulation is that this technology is 
particularly well suited to doing transactional recording and analysis. This is particularly 
true because the computer can record, as well as analyze, in a completely rigorous manner. 
Since transactional recording/analysis is, of necessity, an algorithmic process, its application 
by a computer is, by definition, completely explicit and consistent. 

Implementation 

To understand the current practice of how computer technology is applied to instructional 
simulation, it is necessary to understand the distinction between the software which determines the 
behavior of the computer and the simulation database(s) which determines how the instructional 
simulation itself behaves. 

For the purposes of this discussion, software refers to computer programs written in some high 
level programming language such as PASCAL or FORTRAN. These languages are designed for 
expressing the algorithms v/hich govern the computer's behavior. Using these languages effectively 
requires the specialized skills of a trained programmer who can translate the task to be done into the 
types of algorithms that a computer can execute. By this definition, software is computer-oriented. 

On the other hand, a database or set of databases which define the behavior of a computer 
based simulation are human-oriented. This means that the database(s) are designed to express the 
characteristics of the simulation in a form that is meaningful to a simulation content expert. This 
relieves the content expert from any need for skill in using programming languages. 

When a simulation is based upon a suitable algorithmic model, then developing computer support 
for the simulation requires a heavy investment in software/database design and development. For the 
software, this takes the form of: 

defining the problem(s) to be solved. 



reducing those problems to algorithmic definitions. 



coding the algorithms in a suitable computer source language. 

testing and debugging the source code that embodies those algorithms. 

writing the documentation upon which programers will depend for the future support of tne 
source code. 

For the database(s) which define the nature and behavior of the simulation model, design and 
development means: 

determining what types of databases are needed to define the simulation model. 

determining what kinds of data structures are required to embody those databases. 

developing, where necessary, the compiling software to build these data structures. 

debugging these databases so that they produce the desired behavior in the simulation. 

writing the documentation that future scenario designers will need in order to support 
existing scenarios and to create new scenarios. 

Even after the design and development are completed, it is necessary to make an additional long 
term commitment to support both the software and databases which embody the simulation. For the 
software, this means: 

correcting bugs which are discovered through the continued use of the simulation. 

revising the software to utilize new capabilities in the simulation database(s) as they are 
developed. 

revising the software in order to take advantage of new technologies in computer hardware 
and software as they become available. 

revising the software to satisfy perceived needs for changes in the simulation utilities. 
Such utilities include the user interface and automated support for evaluative data 
gathering. 

For the simulation databases, this continued support takes the form of: 

correcting bugs which are discovered through the continued use of the simulation. 




revising the databases to improve their fidelity in model representation. 

writing new scenarios and updating old scenarios as the need and/or opportunity arises. 

In return for the investment of this time and effort, the simulation and its associated repertoire of 
scenarios embodies the expertise which the simulation is designed to convey. 



DESIGN CONSIDERATIONS 

User Interface 

The very fact that a computer is being used to support an instructional simulation means that the 
user(s) of the simulation will have to learn the skills required to use the computer. The notion here 
is that one must know how to manipulate the computer in order to take advantage of the 
instructional simulation. Shneiderman (1986) recognizes this as syntactic (computer manipulation) as 
opposed to semantic (simulation specific) knowledge. To the extent that these syntactic skills are 
irrelevant to the simulation itself, the learning involved may be referred to as unrecoverable as 
opposed to recoverable learning which is relevant directly to the task or event being simulated (Frank 
Mabry, personal communication). Ideally, the cost of this unrecoverable knowledge should be kept as 
low as possible through the use of computer - user interfaces which are carefully and consistently 
designed. 

Pavia (1990) has written an excellent description of the features which constitute a good computer 
based instructional simulation. The example he uses is SQUALOR, which is a simulated exercise in 
organic qualitative analysis. Among the factors which constitute a good user interface for a computer 
simulation are the following: 

1. Interactivity 

The essential notion of interactivity, as described by Pavia (1990), is that the users must 
feel that they are in control and that the simulation is responsive to their needs and goals. 

a. the "home base* concept 

Central to this idea is the concept of a "home base" to which the user knows that 
(s)he can always return. The "home base" is a known starting point from which the 
user can browse through all options offered by the simulation in order to get a feel 
for how the simulation is laid out. At any time, and from any point in the 



simulation, however, the user can return to the "home base" via a consistent 
keypress (usually by pressing the Escape fc-y one or more times. Typically, this 
"home base" is a main menu which leads to all other options in the simulation. 

menu design 

Menu design is another critical feature of interactivity. Pavia (1990) favors drop 
down-menus so long as there are only a few options contained in each menu. Such 
menus are particularly useful when a mouse is available. However, there is also a 
danger that neatly organized and simplified menus could preempt some of the 
organizational thinking that the learners should do for themselves. This issue 
arises when the learner must make a strategic decision when choosing an option 
from the menu. 

In Pavia's SQUALOR program, such a situation arises when the learner must 
choose from among many analytical tests which vary both in power and cost 
(usually in direct proportion to each other). In this case, Pavia uses a large, 
complex menu which provides some sense of the complex choices the organic 
chemist faces when conducting a real analysis. To expedite the use of such a 
complex menu, Pavia provides single key input for selecting menu options. 

Interruptability 

Unless there are good reasons not to do so, it should be possible for the learner to 
interrupt the simulation at any point and store a recallable image of the 
simulation's state variables. This allows the learner to begin a new problem, or 
resume the old problem at will. It also has the advantage of allowing the learner 
to store an image of the simulation at a point when some difficulty arises so that 
the problem can be reviewed by the instructor. 

availability of help 

There should be both generalized help and context-sensitive help available at all 
times. However, when designing the online help for a simulation, it is critically 
important that the help not unduly bias the learner toward conclusions or actions 
which (s)he should draw from their own judgement within the context of the 
simulation. 

self-assessment 

The simulation should provide some feedback to the learner as to "how well they 
are doing". In Pavia's SQUALOR, this takes the form of recording how much of th 
unknown organic sample is consumed by each test as well as charging the learner 
varying number of resource points for each test depending on the difficulty and 



usefulness of the test. Adequate feedback of this type lets the learner evaluate 
whether they have solved the simulation problem efficiently or by brute force. 

f. availability of appropriate information and tools 

Ideally, the learner should not have to bring any extraneous materials to the 
simulation session. If the simulation requires the use of a calculator or note pad, 
these can be provided as pop-up utilities within the simulation itself. Also, any 
reference materials such as data tables or graphical data should be accessible from 
within the simulation. 

2. Visually interesting and informative displays 

a. minimal clutter 

There is a natural tendency for instructors who are used to the printed page to 
cram as much information as possible onto the available display area. The reader 
will frame the information for themselves. On a computer display, the information 
is framed for the user. In this context, excess information is a distraction which 
detracts from the learning experience; therefore, the tendency to cram information 
onto a screen should be resisted when designing displays. 

b. highlighting and emphasis 

The judicious use of contrasting colors helps to highlight important information. 
Also, enclosing critical information within frames and boxes helps focus attention. 
If the learner is to follow a logical sequence of displays, then it is helpful to vary 
the size, format and/or color of the displays to enhance the feeling of moving from 
place to place. 

c. display consistency 

Screen displays should have a "family appearance" to make it easier for the learner 
to keep track of where (s)he is in the simulation. For example, if all help displays 
shared a common color and/or format, then it is easy for the learner to know when 
(s)he is looking at a help display. The same is true of menus, data tables, etc. 



Model-driven vs. Script-driven Simulation 

Simulations can be characterized as "model-driven" or "script-driven". A "model-driven" simulation 
one in which the events that take place, and the way in which they take place, are governed by an 



underlying algorithmic model of the reality being simulated. Typically, the model-driven simulation 
has a random number generator at its heart. It is the output of the random number generator, as 
interpreted by the frequency distributions incorporated into the model, which determines what 
happens in the course of any given simulation run. By contrast, a "script- driven" simulation is one in 
which the course of events is controlled by a predefined script. The script is produced by a content 
expert who specifies the events that are to occur and the sequence in which they occur. A pure 
script-driven simulation is deterministic. 

It is, of course, quite possible for an instructional simulation to follow a model-driven/script-driven 
mixed paradigm. An example of this is the MALOS-OPS (Snellen & Murray, 1985) and MALOS-QDX 
(Snellen & Murray, 1988) simulated combat environments developed for the U. S. Army. In these 
simulations, the combat mechanics follow a model-driven paradigm while the opposing force (OPFOR) 
tactics are script-driven. The degree to which these two design paradigms are used in constructing an 
instructional simulation is a matter of judgement. The simulation designer must decide the mix 
between model-driven and script-driven techniques when constructing a simulation. 



Mode of Learner/Computer Interaction 

Since the art of simulation is based on the notion of creating a self-contained reality in which the 
learner(s) play some role, there are two very obvious strategies by which the simulation designer can 
employ computer support. The computer may act as a participant in the simulation reality or it can 
act as an non-participating mediator in that reality. 

1. Computer as Participant 

Actually, it would be more appropriate to speak of the programmer and/or scenario 
designer as being the participant since that is the true source of intelligence which drives 
the role played (or perhaps more accurately, executed) by the computer. Normally, for the 
computer to act in the role of participant, the simulation must have, at least in part, an 
underlying model-driven design paradigm. 

2. Computer as Mediator 

By definition, a computer based simalation will always include this role as a function 
served by the computer. That is, the computer will manage communication between the 
learner(s) and the simulation reality as well as all of the routine internal housekeeping 
tasks of data management, disk I/O, and whatever else is necessary just to make the 
simulation operate. The real issue is whether or not these constitute the only tasks 
performed by the machine. The classic simulation paradigm in which the computer acts 
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only as mediator is when the simulation is based on the interaction between multiple 
human participants. 

The reason why the distinction between participant/mediator roles for the computer is important is 
that it has a profound impact on the design strategies that can be used by the simulation system 
designer. It has a significant impact, as well, on the way in which the resulting instructional 
simulation can be used. If the computer is to act as participant, then the domain of interactions 
possible is constrained by the depth and flexibility of the underlying model upon which the computer 
depends in order to execute its role. On the other hand, such a design paradigm allows the 
possibility of solitaire play in which the learner has the maximum degree of control over his/her 
learning environment. If the computer is to act only as mediator, then the simulation reality can 
have all of the depth and flexibility that is inherent in purely human interactions. However, this 
precludes any meaningful way to operate the simulation in solitaire mode. 



Database Oriented Simulation 

It has become generally recognized that it is preferable to embody an instructional simulation 
within a database rather than in the software that runs the simulation. This is what Pavia (1990) 
refers to as "Adequate Practice' 1 ; that is, providing the means by which the simulation can readily be 
changed without altering the software that uns the simulation. Typically, the software is nothing 
more than a central driver that reads a set of one or more database files which define a scenario 
within the simulation. The software then runs the simulation according to the terms and 
specifications found in the database. In Pavia's SQUALOR, the database consists of a data record 
indicating the properties of the organic compound to be identified along with specifications as to the 
results of any analytical procedures that the learner might wish to perform. Each database, then, 
constitutes a scenario within the instructional simulation. 

What apparently has not yet become generally recognized, is the best means by which database 
(scenario) development can be supported. Usually, this support takes the form of a series of database 
editors which are used to construct and maintain the database(s) used by the simulation drivers). 
The problem (as is noted by Pavia) is that the design, development and support of these database 
editors usually is as costly as it is for the simulation driver itself. Moreover, if the scenario database 
is at all complex (as it is in MALOS-OPS and MALOS-QDX), then database construction is bound to 
be error prone with no good way of doing internal error checking. 

Some types of database construction do, in fact, lend themselves readily to the use of a database 
editor. For example, when the data is graphic (or more generally when there is a spatial and/or 
temporal aspect to the data), then an editor may provide the best means of creating the necessary 
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database. In the MALOS system, this was the approach taken to create both the OPFOR tactical 
plans and the terrain maps on which those plans are executed. On the other hand, we feel that all 
quantitative and textual data is far better handled by recording this data in a text file. A compiler 
can read the text file(s) and build the necessary databases (Mabry, Personal Communication). This 
approach provides a number of significant advantages as follows: 

1. A database compiler can be developed more easily than can a database editor. The 
compiler does not have to incorporate the sophisticated insert, delete, modify and data 
display capabilities that are required in a database editor. In effect, these functions are 
already available in whatever text editor one chooses to use when writing the text files 
that become the source files for the database compiler. 

2. Making extensions to a database compiler, as the needs of the simulation grow, is faster 
and easier than making extensions to a database editor for the same reasons cited under 
#1. 

3. When using a database compiler, the source files become the main depository of the 
database library. Because these contain the data in a form easily readable by a human, it 
is not necessary to write a separate print facility to produce hard copy from a database 
maintained by a database editor. 

4. If the simulation database ever becomes corrupted or lost, it is only necessary to recompile 
it if one has a database compiler. Otherwise, it is necessary to reenter the database by 
hand using a database editor. 

5. It is easier to incorporate database consistency checking into a database compiler than in a 
database editor. 

The Problem of Evaluation 

The value of an instructional simulation is unclear unless some means exist to evaluate its 
effectiveness. This is particularly true since instructional simulation tends to emphasize the 
development of soft skills as defined by Begland (1982). While there is no substitute for human 
judgement in evaluating any instructional method, computer technology offers some unique advantages 
in constructing an evaluation strategy. First, the rigor with which a computer based instructional 
simulation must be designed provides an excellent basis for developing a well disciplined data 
collection format. Second, a computer is superbly well suited to perform transactional recording 
during the simulation run. From the transactional record, it is possible, at a later time, to 
reconstruct the simulation run for analysis and evaluation. This approach was used in. the debriefing 



option of the MALOS-OPS and MALOS-QDX combat simulation systems (Snellen & Murray, 1985, 
1988). Finally, becauw these computer generated transactional records have a consistently defined 
format, it is easy to use the computer to automate the statistical analysis of the transactional records. 
This was the approach used in a study of problem solving behavior in which one of us (J. Snellen) 
provided computer programming support (Steinberg et al. 1986). 



GLOSSARY 



Algorithm 

An algorithm is any explicitly defined procedure for accomplishing a task. The procedure is laid 
out as a detailed series of steps to be followed in sequence. When the sequence of steps is 
completed, the objective of the task will be reached. 

Compiler 

A compiler is an automated language translator. It reads messages written in human readable 
source language and translates them into a machine executable object language. Normally, 
the source language message is contained in an input file which is read by the compiler. The 
object language produced by the compiler is contained in an output file. The source language 
can be a formal programming language such as PASCAL or FORTRAN, or a simulation-specific 
database language. 

Database 

A database is any body of information written in a predefined ordered format. Under normal 
usage, the term database generally refers to a body of information that has been assembled by 
humans in a form that can be read and manipulated by a machine. Common types of databases 
are inventories, mailing lists, and pricing tables. However, a database can also be a table of 
values which govern how a computer program will behave. A database can also be a script 
which governs how an instructional simulation will behave through time. 

Debugging 

Algorithms, as well as the way those algorithms are expressed in source language frequently 
contain mistakes. These mistakes produce unexpected (and usually unwanted) results when the 
algorithm is executed. Early in the history of Computer Science, the term bug was adopted as a 
name for these mistakes; hence the process of finding and removing these mistakes became known 
as debugging. 

Disk I/O 

Computers store information on magnetic media which are usually in the form of floppy or hard 
disks. To be useful, that information must be transmitted from the disk into the computer as well 
as from the computer out to the disk. The terms used to describe this two-way transfer of 
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information are input for information moving from the disk into the computer and output for 
information movi ig from the computer out to the disk. Normally, this is abbreviated as disk I/O. 

Hardware 

This term refers to any piece of physical machinery. It is called "hard" because once it is 
constructed, its form is not readily altered (also, see the definition of software). 

Input Pile 

A computer program operates on data supplied to it from outside the program. An interactive 
program normally receives data typed by a human user using a keyboard or mouse. However, 
such data also can be placed in a disk file from which the program can read its input data. Such 
a file is called an input file. 

Object Language 

Humans and computers understand two very different types of languages. Programming languages 
(such as PASCAL and FORTRAN) are designed to express formal algorithms in a way that 
humans can write and understand readily. However, the only language that computers 
understand is a sequence of raw numbers in which each number indicates the next instruction to 
be executed or evaluated. When a human needs to communicate with a computer, the human 
writes the message in a form that is understandable to him/her (for writing programs, this means 
using one of the programming languages; for describing data, this means writing the data in some 
database language). Since this human-understandable language is the source of the information 
given to the computer, it is referred to as the source language. A compiler then translates this 
source language into an object (i.e. a sequence of numbers that specify the machine instructions 
to be executed) that the computer can "understand". These objects are then referred to as the 
object language. 

Also, see the explanation under compiler. 
Operating System 

As a raw piece of hardware taken out of the shipping crate, a computer is virtually useless. All of 
the components such as disk drives, keyset, monitor, communication ports, central processing unit 
and memory are there; however, to do anything useful, one needs to make all of these hardware 
components work together in a meaningful way as well as communicate with the human user 
and/or with other machines. These goals are accomplished by providing the computer with a set 
of programs which allow it to operate in a coherent manner. This set of programs ifi known 



collectively as the operating system. For personal computers, the best known operating 
systems currently are Microsoft's DOS, Apple's MultiFinder and UNIX. 

Output File 

When a computer program produces data that is to be transmitted to some location outside itself, 
that data is called output. An interactive program normally transmits output data to the 
computer's display screen, or some other hardware device. This data also can be written to a disk 
file for storage and later use. Such a disk file is called and output file. 

Pop-up Utility 

This term applies specifically to the DOS operating system which is designed to run one task at a 
time. It often happens that one needs to use some utility, such as an appointment calendar or a 
calculator while they are running a program (such as a spreadsheet). In a single task operating 
system such as DOS, it normally is necessary to stop one program before invoking another. One 
way around this is to write a utility in such a way that it can be loaded into the computer's 
memory prior to starting the main program. The utility can then be activated by pressing some 
special key which temporarily halts the main program and begins running the utility program. 
When the utility program is activated in this way, it "pops up" temporarily on the screen. 
Therefore it is referred to as a Pop-Up Utility. 

Single Key Input 

A menu is a natural decision point in a computer program. It allows the user to control what the 
program does next. If the menu is used frequently, then the programmer usually will design the 
menu to operate via single key input. This means that one letter (usually the first letter) of 
each menu choice is highlighted to indicate that it is the active letter for that choice. When the 
user presses the key for that letter, the menu immediately invokes the function associated with 
that choice. Note that the user does not have to press the Enter (or Return) key, only the 
highlighted key. If the menu is frequently used, then the user soon memorizes which keys 
correspond to which menu choices. This allows the experienced user to operate the menu with 
maximum efficiency. 

Scenario 

This term has essentially the same meaning in instructional simulation as it does in drama. A 
scenario is a description of the particular details of the simulation. For a role-playing simulation, 
the scenario defines the roles to be played, as well as the goals, problems, and viewpoint of each 
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role. In computer based simulation, this term refers to the database which defines the nature of 
the simulation. 



Software 



Central to the notion of a stored program computer is the idea that the function of the machine is 
not determined by the hardware (i.e. the physical machine itself). Rather, the function that the 
machine serves is determined by a stored, executable body of software. For example, a single 
piece of hardware (e.g. and IBM AT) can be a word processor, a spreadsheet calculator or a 
database manager depending on type of software that the hardware currently has loaded and is 
executing. From the programmer's perspective, software is called "soft" because it is easily 
modified to suit changing needs and objectives. 



Source Language 



See the explanation under object language 



State Variable 



A computer based simulation (or, for that matter, any computer program) operates by keeping 
track of, and manipulating, a series of numerical values. Some of these values serve the purpose 
of recording the state of the simulation. During a wargame, for example, there may be variables 
for recording the number, type and location of combat units, the supply state of these units, the 
current weather conditions, etc. Because these variables determine the current state of the 
simulation, they are referred to as state variables. 
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PCSL/IDOC Task Order 90-3 a 
Interview Form 



Interviewee Name: 



Date of Interview: 



Phone: 



Issue #1 (Situational Audit): 

We would like to establish your working situation; that is, the context in which you 
have developed and used an instructional simulation. Could you summarize this for 
us? 



Issue #2 (Relationship to use of simulation) 

What work have you done with instructional simulation? Do you design it, construct it, 
publish about it? 



Issue #3 (Design of instructional simulation) 

What, in your judgement, is the best approach to designing instructional simulation? 



Issue #3 (Suggestions) 

Do you have any suggestions/comments regarding use of instructional simulation in 
your class(es). Are there any problems/drawbacks in using instructional simulation? 



Issue #5 (References) 

Can you think of anyone else that we should interview about instructional simulation? 



Issue #6 (Call Back) 

May we call back if there are any further questions we would like to ask? 



This questionnaire was used m a guide for discussion during the interviews rather than as a rigid set of questions to be 
answered. Hie interviews were conducted according to an expert interview model in which the interviewee was encouraged to 
discuss their field in an open-ended manner within tne parameters set by these questions. 
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PCSL/IDOC Task Order 90-3 
Interview Report 



Interviewee Name: Bill Carstensen 

Date of Interview: December 14, 1990 



Dr. Orstensen is Associate Professor of Geography at Virginia Polytechnic Institute and State 
College (informally known as Virginia Tech.). He has been with the Department of Geography 
since 1983. His area of teaching is primarily in cartography, with sophomore courses in map 
reading, a course at the junior level in map production, courses at senior and graduate level in 
computer cartography and a graduate level course in cartographic information systems. 

Dr. Carstensen received his Bachelor's degree with honors in Geography in 1976, his Master's 
degree in Cartography and Computer Mapping in 1978 and his Ph.D. in Automated Mapping 
in 1981 from the University of North Carolina at Chapel Hill. 

From 1981 to 1983 he was at the University of Wisconsin at Green Bay as an assistant 
professor. 

He has some formal training in Computer Science. 

His research area is primarily in the area of accuracy of spatial databases for geographic 
information systems and for computer cartography. He is also interested in cartographic 
education in which he has done some simulation work. 

He has developed a computer based simulation in map orienteering which is he calls 
"surrogate travel" in that the user of the software package experiences a simulated trip 
through unknown terrain. The sport of orienteering consists of placing a series of markers in 
and area which includes a variety of different types of terrain. The locations of the markers 
are indicated on a map. The object is for the onenteer to traverse the designated course, 
checking in at each marker and then reaching the finish line in the shortest time possible. 
The emphasis in the sport is on good map reading and judgement. Dr. Carstensen uses this 
simulation in some of the classes he teaches. 



PCSL/IDOC Task Order 90-3 
Interview Report 



Interviewee Name: John Shatzer 

Date of Interview: December 14, 1990 



John Shatzer holds a Bachelor's degree in psychology (1967) at University of Evansville in 
Indiana, a Master's degree in Education (1969) at Indiana University. John currently is 
working on a Ph.D. in educational psychology at University of Illinois. John works for the 
University of Illinois College of Medicine as an instructional coordinator for the clinical phase 
of the curriculum. His office is at Carle Hospital in Urbana, Illinois. 

John helps design live (i.e. role playing) patient simulations for use in the College of Medicine 
clinical program. The purpose of tne simulations is to help medical students learn how to take 
medical histories. The medical history simulations are used both for training and evaluation of 
the student. The simulations are used for training and evaluating medical residents as well as 
students. The medical students which go through these simulations are in their second year of 
training when they are beginning to learn the clinical environment. 
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PCSL/IDOC Task Order 90-3 
Interview Report 



Interviewee Name: Charles Herring 

Date of Interview: December 17, 1990 



Charles Herring received a B.S. in Physics from the University of Mississippi in 1981 and a 
M.C.S. (Masters of Computer Science) from the University of Illinois in 1986. He has worked 
for the Construction Engineering Research Laboratory (CERL) in Champaign, Illinois for the 
past 10 years. During that time he has worked on applications development in construction 
management and tactical command and control systems. For the past three years, his primary 
interest has been in the development of simulation systems in operations research. He is 
rimarily a software specialist currently working on a microcomputer version of the Force 
tructure Tradeoff Analysis Model (FSTAM). This combat simulation is intended to be an 
evaluative tool for the role of combat engineering in battle. Charles' interest is in the 
application of object oriented programming in simulation design. In addition, he is interested 
in the problem of machine representation of real numbers in developing computer based 
simulations. 



PCSL/IDOC Task Order 90-3 
Interview Report 



Interviewee Name: Bill Olson 

Date of Interview: December 18, 1990 



Dr. Olson holds both a Master's and Ph.D. in pharmacology. He has three years experience as 
a post doctoral fellow in clinical pharmacology. He also holds a Master of Computer Science 
degree at U of I. He currently holds the position of Computer Analyst for Academic 
Applications and Assistant Professor of Clinical Veterinary Medicine at the School of 
Veterinary Medicine at the University of Illinois. 



PCSL/IDOC Task Order 90-3 
Interview Report 

Interviewee Name: Marie Amone 

Date of Interview: January 14, 1991 



Ms. Arnone studied English at the University of London in the early 1970's and then received 
a M.B.A. from New York University in 1981. She works as an independent consultant in 
experienced based learning for New York University in collaboration with Steven Stumpf and 
David Kolb. Dr. Stumpf and Dr. Kolb are professors of Business Management at the NYU 
School of Business. She currently is helping to develop a business simulation for use at the 
Defense Systems Management College. 
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